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SPECIFICATION 

MOVEMENT DECISION METHOD FOR CAPTURING SUB-PIXEL MOTION 
IMAGE SUITABLE FOR SUPER-RESOLUTION PROCESSING, AND IMAGE 
CAPTURING DEVICE USING THE SAME 

TECHNICAL FIELD 

The present invention relates to super-resolution 
technique for generating one high resolution image from plural 
low resolution images, and particularly, relates to movement 
decision method of image capturing object or image capturing 
element for capturing two-dimensional sub-pixel motion image 
suitable for super-resolution processing, and image capturing 
device using this movement decision method. 

BACKGROUND TECHNIQUE 

It is necessary to capture plural images in which the 
displacement between images are two-dimensionally distributed 
sufficiently closely at an equal interval as much as possible 
so as to perform the super-resolution processing for 
generating one high resolution image by utilizing plural input 
images of low resolution. However, it was difficult to capture 
the plural images having the two-dimensional displacement 
suitable for such super-resolution processing. 

Therefore, there were conventionally many cases 
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adopting the following methods. 

(a) The image capturing object is captured while the 
image capturing element (e.g., CCD image capturing element) 
within the image capturing device (e.g., digital CCD camera) 
is two-dimensionally finely moved by a known moving amount. 

(b) Sufficient many images are manually captured 
without fixing the image capturing device (e.g., digital CCD 
camera) to a tripod, etc., and the image which have displacement 
suitable for the super-resolution processing is selected from 
the many captured images. 

However, in the conventional method of the above (a) , 
there are disadvantages in that two actuators are required to 
perform a two-dimensional operation in the image capturing 
device (e.g., digital CCD camera), and a moving mechanism of 
the image capturing device (e.g., CCD image capturing element) 
becomes complicated so that the image capturing device becomes 
an expensive product. 

The conventional method of the above (b) is a method for 
expecting an accidental displacement by a portable camera, etc. 
Therefore, problems exist in that there is no guarantee able 
to capture the displacement image required in the 
super-resolution processing, and it is not known to capture 
the image till when. Namely, when the image capturing device 
(e.g., digital CCD camera) is manually moved, there is a 
possibility that the degree of motion freedom is increased. 
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Therefore, it is impossible to voluntarily control the 
displacement amount less than a pixel unit (hereinafter simply 
called a sub-pixel) . Accordingly, there is a difficulty in 
that no super-resolution image of high quality can be 
necessarily obtained. 

The present invention has been made in view of the 
circumstances described above. An object of the present 
invention is to provide a movement decision method for 
determining direction and distance of one-dimensional 
movement of image capturing object or image capturing element 
for capturing two-dimensional sub-pixel motion image suitable 
for super-resolution processing, and an image capturing device 
using this movement decision method. 

DISCLOSURE OF THE INVENTION 

The present invention relates to a movement decision 
method for capturing sub-pixel motion image suitable for 
super-resolution processing. The above object of the present 
invention is effectively achieved by the construction that, 
while an image capturing object is moved along a 
predetermined one-dimensional moving direction, a sequential 
image of said image capturing object is captured by a fixed 
image capturing device, said sequential image is set to a 
two-dimensional sub-pixel motion image suitable for 
super-resolution processing, and said one-dimensional moving 
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direction of said image capturing object in a coordinate system 
normalized by aspect ratio of pixel of an image capturing 
element within said image capturing device is determined to 
p/q of a rational number, wherein: one pixel of vertical 
direction of said coordinate system is divided by an integer 
p, and one pixel of horizontal direction of said coordinate 
system is divided by an integer q. 

The above object of the present invention is also 
effectively achieved by the construction that absolute value 
of said integer p and absolute value of said integer q are 
integers which are not small integers. The above object of 
the present invention is also effectively achieved by the 
construction that a moving direction for setting value of an 
evaluation function Cover (Lact) to be smaller than 1 is set 
to said one-dimensional moving direction of said image 
capturing object when magnification of said super-resolution 
processing is fixed and known. The above object of the present 
invention is also effectively achieved by the construction 
that a moving direction for setting value of an evaluation 
function SCover(LM) to be smaller than 1 is set to said 
one-dimensional moving direction of said image capturing 
object . 

The present invention also relates to an image capturing 
device using the movement decision method for capturing 
sub-pixel motion image suitable for super-resolution 
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processing. The above object of the present invention is also 
effectively achieved by an image capturing device which 
captures a two-dimensional sub-pixel motion image suitable for 
super-resolution processing, said image capturing device 
comprises : a driving mechanism which one-dimensionally drives 
an image capturing element within said image capturing device 
along an image capturing element moving direction, wherein: 
said one-dimensional moving direction of said image capturing 
object determined by the method according to the present 
invention is set to said image capturing element moving 
direction, a sequential image of a fixed image capturing object 
is captured while said image capturing element is moved by said 
driving mechanism along said image capturing element moving 
direction, and said sequential image is set to said 
two-dimensional sub-pixel motion image. 

The above object of the present invention is also 
effectively achieved by an image capturing device which 
captures a two-dimensional sub-pixel motion image suitable for 
super-resolution processing, said image capturing device 
characterized by arranging a member for optically moving image 
in a predetermined direction between a lens and an image 
capturing element, wherein: said one-dimensional moving 
direction of said image capturing object determined by the 
method according to the present invention is set to said 
predetermined direction, a sequential image of a fixed image 
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capturing object is captured, and said sequential image is set 
to said two-dimensional sub-pixel motion image. 

BRIEF DESCRIPTION OF THE DRAWINGS 

Fig. 1 explains pixel of CCD image capturing element 
having rectangular shape. 

Fig. 2 is a typical view showing image capturing 
landscape of sub-pixel motion image suitable for the 
super-resolution processing by a fixed image capturing device 
in the present invention. 

Fig. 3 is a conceptual view showing one-dimensional 
movement of image capturing object in the present invention. 

Fig. 4 explains the relation between distance between 
sub-pixel tracks A and q in the- present invention. 

Fig. 5 is a view showing the relation between distance 
between sub-pixel tracks A and moving direction of image 
capturing object. 

Fig. 6 is a graph showing the relation between distance 
between sub-pixel tracks A and moving distance L. 

Fig. 7 shows the relation between maximum value Amax 
(Lact) of distance between sub-pixel tracks A and moving 
direction, and the relation between the difference between 
maximum value and minimum value of distance between sub-pixel 
tracks A (Amax (Lact ) -Amin (Lact ) ) and moving direction. 

Fig. 8 explains evaluation function Cover (Lact) . 
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Fig. 9 shows numerical calculation algorithm for 
calculating Amax(Lact) and Amin(Lact). 

Fig. 10 explains optimum sub-pixel motion with respect 
to a minimum number of captured image required in the 
super-resolution processing of predetermined magnification. 

Fig. 11 shows the relation between evaluation function 
SCover(LM) and moving direction a in the case of different 
maximum moving distance LM. 

Fig. 12 explains that distance between sub-pixel tracks 
is gradually reduced as moving distance Lact of image capturing 
object is increased when moving direction is appropriately set 
on the basis of evaluation function SCover(LM). 

Fig. 13 shows sub-pixel distribution when image 
capturing object is moved at a constant moving speed. 

Fig. 14 shows sub-pixel distribution when moving speed 
of image capturing object is changed in accordance with time 
and image capturing object is moved at moving speed gradually 
slowed . 

Fig. 15 shows experimental result of the 
super-resolution processing when moving direction is 
appropriately set . 

Fig. 16 shows experimental result of the 
super-resolution processing when moving direction is 
inappropriately set . 

Fig. 17 shows a constructional example for realizing 
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one-dimensional movement of image capturing element within 
image capturing device. 

Fig. 18 shows another constructional example for 
realizing one-dimensional movement of image capturing element 
within image capturing device. 

Fig. 19 shows a constructional example for realizing 
change of image capturing direction of image capturing device. 

Fig. 20 shows a constructional example for realizing 
one-dimensional movement of moving stage. 

Fig. 21 shows a constructional example for realizing 
movement of an image by using an optical system within image 
capturing device. 

BEST MODE FOR CARRYING OUT THE INVENTION 

Best modes for carrying out the present invention will 
next be explained with reference to the drawings. 

In the present invention, an aimed point is to construct 
an optimum two-dimensional sub-pixel motion distribution used 
in the super-resolution processing from sequential images in 
which one-dimensional movement of image capturing object is 
captured by an image capturing device (e.g., a digital camera 
using a CCD image capturing element and hereinafter also simply 
called a CCD camera) . Namely, in the present invention, 
two-dimensional sub-pixel motion image suitable for the 
super-resolution processing is constructed from sequential 
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images in which one-dimensional movement of image capturing 
object is captured. In this case, the most important 
characteristic of the present invention is that 
two-dimensional sub-pixel motion image suitable for the 
super-resolution processing is obtained by determining 
direction and distance of one-dimensional movement of image 
capturing object by using an evaluation function described 
later . 

In short, when track of one-dimensional movement of image 
capturing object (hereinafter also called a moving track) is 
mapped to an area of 1 X l [pixel] , the most important 
characteristic of the present invention is that direction and 
distance of one-dimensional movement of image capturing object 
are determined by utilizing evaluation function representing 
that moving track "covers" this entire area of IX l [pixel]. 
Term "pixel" used in the present invention (hereinafter also 
called picture element) means pixel of image capturing element 
(e.g., CCD image capturing element) within image capturing 
device . 

Here, the following conditions are set as preconditions. 

(1) Moving track of image capturing object is started 
from origin (0,0) of coordinate system on an image. 

(2) With respect to direction of one-dimensional 
movement of image capturing object 0 (hereinafter also simply 
called moving direction 8 or moving angle 0) , O°<0<45° is 
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considered. Namely, 0<a<l is formed with respect to gradient 
of straight line representing moving direction a (hereinafter 
also called gradient of moving direction a) . With the 
exception of O°<0<45°, moving direction having other angle 0 
becomes equal to above 0°<9<45° by performing appropriate 
coordinate transformation such as replacement of horizontal 
direction variable and vertical direction variable, etc. 

For brevity of explanation, there is also a case in which 
gradient of moving direction is simply called moving direction 
in the present invention. For example, when gradient a of 

moving direction 0 is 1/4, it can be also said that moving 
direction a is 1/4. 

(3) Distance of one-dimensional movement of image 
capturing object (hereinafter also simply called moving 
distance, realistic moving distance, or actual moving distance 
of image capturing object) is set to horizontal direction 
distance on an image. Accordingly, as gradient a of moving 
direction 0 approaches 0, the difference between actual moving 
distance of image capturing object along moving direction and 
distance of one-dimensional movement of image capturing object 
is reduced. Accordingly, in the present invention, when 
plural moving directions suitable for the super-resolution 
processing exist, it is desirable to select moving direction 
in which gradient a is close to 0. 

(4) Moving direction 0 of image capturing object is 
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considered in a coordinate system normalized in aspect ratio 
of pixel of image capturing element (e.g. , CCD element) within 
image capturing device (e.g., CCD camera). Namely, the 
coordinate system normalized by the pixel, i.e., sampling 
interval is also considered in an image except for 1:1 in 
longitudinal and transversal ratios. 

Namely, no picture element (pixel) of CCD image capturing 
element necessarily has a square shape. For example, as shown 
in Fig. 1, picture element (pixel) of CCD image capturing 
element can also have a rectangular shape. When moving 
direction of image capturing object is 45° with respect to pixel 
of CCD having rectangular shape, sub-pixel track (definition 
of sub-pixel track is described by <1>) is different from the 
case of pixel of CCD having square shape. Accordingly, in the 
present invention, it is considered in the coordinate system 
normalized by aspect ratio of pixel of CCD image capturing 
element. Namely, in the case of Fig. 1, direction of 45° 
provided when it is considered in units of x-axis and y-axis 
shown in the coordinate system is set to moving direction of 
image capturing object. 

<1> Basic characteristic of sub-pixel track 
Next, on the premise that track of image capturing object 
linearly moved can be precisely measured, basic characteristic 
relating to a sub-pixel component included in this moving track 
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will first be described. In short , when the premise of moving 
image capturing object at an unlimited length is set to a 
precondition, the corresponding relation between moving angle 

(moving direction) of image capturing object and distance 
between sub-pixel tracks will be considered as follows. 

Here, Fig. 2 is a typical view showing image capturing 
landscape of sub-pixel motion image suitable for the 
super-resolution processing by a fixed image capturing device 

(e.g., a CCD camera) in the present invention. As shown in 
Fig. 2, a moving mechanism (e.g., a moving stage ) is constructed 
so as to be inclined and moved straightly (i.e., 
one-dimensionally) with respect to the fixed image capturing 
device (e.g. , CCD camera) . The image capturing object is fixed 
to this moving mechanism (here, the moving stage) . The moving 
stage (image capturing object) moved one-dimensionally along 
a predetermined moving direction is captured by the image 
capturing device in time series. 

Conventionally, in order to capture sub-pixel motion 
image suitable for the super-resolution processing (i.e., an 
image having an appropriate two-dimensional displacement 
between images) , a slight displacement is made in vertical 
direction and horizontal direction by combining two moving 
stages. In the present invention, when image capturing object 
is moved in a direction inclined with respect to fixed image 
capturing device, i.e. , when moving stage is moved in inclined 



12 



direction in the case of Fig. 2, image capturing object is moved 
in vertical direction and horizontal direction. Accordingly, 
if only appropriate moving direction (moving angle) and moving 
distance are determined with respect to image capturing object , 
it is possible to simply capture sub-pixel motion image (i.e., 
an image having an appropriate two-dimensional displacement 
between images) able to perform the super-resolution 
processing of very high precision. 

Fig. 3 is a conceptual view showing one-dimensional 
movement of image capturing object in the present invention 
as shown in Fig. 2. In Fig. 3, x-axis is an axis representing 
horizontal direction with respect to image capturing element 
(here, CCD element) within fixed image capturing device (here, 
CCD camera) . Y-axis is an axis representing vertical 
direction with respect to image capturing element (here, CCD 
element) within fixed image capturing device (here, CCD 
camera) . Both units of x-axis and y-axis are set to pixel, 
and one pixel here represents one pixel of image capturing 
element (here, CCD element) . 

As shown in Fig. 3, when moving direction (moving angle) 
of image capturing object is set to 0, moving track of image 
capturing object passes along the direction not represented 
by pixel unit but represented by sub-pixel unit in accordance 
with set moving direction 0. 

When the super-resolution processing is performed, the 
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movement in pixel unit is the same if corresponding pixel number 
is shifted. Therefore, only moving amount in sub-pixel unit 
becomes a problem. 

In the present invention, only moving amount in sub-pixel 
unit is considered. When image capturing object is moved in 
moving direction 0 as shown in Fig. 3 and moving track of 
sub-pixel of its image capturing object is collected, this 
moving track is provided as shown in lower portion of Fig. 3. 
In this connection, as shown in Fig. 3, in the case of moving 
angle of image capturing object moved rightward by five pixels 
in x-axis direction and moved upward by two pixels in y-axis 
direction, sub-pixel track ought to be formed as shown in lower 
portion of Fig. 3. An object of the present invention is to 
consider whether an appropriate sub-pixel track is obtained 
by taking which moving angle. 

Here, measuring result of displacement between images 
at observing time of a sufficiently large moving distance is 
considered. The track passing a 1X1 [pixel] area constitutes 
a specific pattern in accordance with moving direction. The 
track with respect to this 1X1 [pixel] area is called a 
sub-pixel track in the present invention. One pixel area 
constructed by four points of (0,0), (1,0), (1,1) and (0,1) 
in coordinate system is considered as this 1X1 [pixel] area. 

Here, if the displacement between images is measured over 
a sufficiently long moving distance, it can be considered that 
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sub-pixel track becomes an equal interval. In this 
supposition, it is premised that gradient a of moving direction 
9 is represented by a rational number. When gradient a of 
moving direction 9 is an irrational number, no sub-pixel track 
becomes an equal interval even when the measurement is 
continued over a long distance. However, since measurement 
precision is finite in actual measurement, sub-pixel track can 
be actually considered as equal interval in measurement with 
respect to a sufficiently long distance. At this time, 
vertical distance between sub-pixel tracks A (hereinafter also 
simply called the distance between sub-pixel tracks) (see Fig. 
3) is considered. 

Moving track of image capturing object is set to y=ax 
(a is a real number) . Gradient a of moving track at this time 
is a measured value able to be calculated by the measurement 
using observed sequential images. While image capturing 
object is fixed, the same result can be also obtained by moving 
an image capturing system ( image capturing element of image 
capturing device) in gradient a with respect to horizontal 
direction of the image. 

Gradient a of this real number is approximated by a 
rational number q/p (here, p and q are respectively integers) . 
Concretely, in consideration of valid digit number of real 
number a, p and q are replaced as follows. 
[Equation 1] 
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p=10 



q«aX10 n 



At this time, moving track can be represented as in the 
following equation 2 by using a greatest common divisor 
gcd(p,q) of p and q. 
[Equation 2] 



Both P and Q in above equation 2 are integers and cannot 
be reduced any more. It can be considered that gradient of 
moving track is 1/P by replacing position x of horizontal 
direction with (Qx) . This gradient 1/P corresponds to that 
one pixel of vertical direction is divided by integer P (see 
Fig. 4). Accordingly, the distance between sub-pixel tracks 
A is provided as in the following equation 3. 
[Equation 3] 



Fig. 4 explains the relation between the distance between 
sub-pixel tracks A and q. Here, q indicates what portions one 
pixel of horizontal direction is repeatedly divided into. In 
the case of Fig. 4(A), the distance between sub-pixel tracks 
A is 1/3, and q is 1. On the other hand, in the case of Fig. 
4 (B) , the distance between sub-pixel tracks A is 1/3, and q 




(Qx) 




gcd(p.g) 



P 
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is 2. As can be seen from Fig. 4(B), one pixel of horizontal 
direction is divided into two. Accordingly, it is well known 
that q does not relate to the distance between sub-pixel tracks 
A. 

As the distance between sub-pixel tracks A is shortened, 
a fine sub-pixel motion can be captured. Accordingly, a 
further good result is also obtained with respect to the 
super-resolution processing using this fine sub-pixel motion. 

Fig. 5 is a view showing the relation between the distance 
between sub-pixel tracks A and moving direction of image 
capturing object. As can be seen from Fig. 5, when moving 
direction of image capturing object, i.e. , gradient a of moving 
track is finely examined in a range from 0 to 1, there are many 
gradients ( moving direction of image capturing object) in 
which the distance between sub-pixel tracks A becomes very 
small . 

However, a large moving distance L shown by the following 
equation 4 is required to realize such a small distance between 
sub-pixel tracks A. For example, in the case of A=0.05 
[pixel] , it is necessary to set moving distance L to 20 pixels. 
[Equation 4 ] 

L = P = — ^— = 1 
gcd{p,q) A 

When the relation between the distance between sub-pixel 
tracks A and moving distance L is shown by a graph, this graph 
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is provided as shown in Fig. 6. As can be seen from Fig. 6, 
the number of effective moving direction in a range until degree 
of realistic moving distance L=30 [pixel] is very slight (5 
to 6) . Namely, gradients of effective moving direction 
q/p=l/40, 1/32, 1/25, 1/20 and 1/16 are formed. 

<2> Evaluation by using maximum value and minimum value 
of the distance between sub-pixel tracks A 

As mentioned above, it is well known that the 
characteristic of sub-pixel component (i.e., sub-pixel track) 
included in moving track of image capturing object is basically 
determined in accordance with moving distance of image 
capturing object. Thus, it is known that it is also necessary 
to increase moving distance of image capturing object in 
accordance with an increase in magnification for performing 
the super-resolution processing. However, it is 

realistically not considered as an actual problem to capture 
a limitlessly long moving track. Accordingly, the 

characteristic of sub-pixel track will be considered as 
follows when it is limited to moving distance of image capturing 
object of length of a certain extent. 

In U <1> Basic characteristic of sub-pixel track", it 
is considered on the premise that one pixel area is "covered" 
at uniform distance between sub-pixel tracks A. However, in 
reality, it is not necessary to "cover" one pixel area at 
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uniform distance between sub-pixel tracks A. Therefore, 
method for evaluating that passing density of sub-pixel track 
is close to uniformity will be considered. 

As mentioned above, the distance between sub-pixel 
tracks A is clearly changed in accordance with moving direction 
(gradient of moving direction) q/p, and actual moving distance 
Lact of observed image capturing object. Here, maximum value 
Amax of the distance between sub-pixel tracks A and its minimum 
value Amin are evaluated as function of moving direction and 
observed moving distance Lact. In reality, the maximum value 
Amax (Lact) of the distance between sub-pixel tracks A and the 
difference between the maximum value and the minimum value 

(Amax (Lact ) -Amin (Lact) ) are evaluated (see Fig. 7). Fig. 7 
shows graphs in which moving distance of image capturing object 

(Lact) is respectively 10 [pixel] , 20 [pixel] and 30 [pixel] 
Figs. 7(A) and 7(B) show graphs in which the moving 
distance Lact is 10 [pixel] . Fig. 7(A) is a view showing the 
relation between the maximum value Amax (Lact) of the distance 
between sub-pixel tracks A and moving direction of image 
capturing object when image capturing object is actually moved 
by 10 pixels in horizontal direction. Fig. 7 (A) is also a graph 
showing the relation between the difference between the 
maximum value and the minimum value of the distance between 
sub-pixel tracks A (Amax (Lact ) -Amin (Lact ) ) and moving 
direction of image capturing object when image capturing 
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object is actually moved by 10 pixels in horizontal direction. 
Fig. 7(B) is a graph for summarizing two graphs of Fig. 7(A) . 
Similarly, Figs. 7(C) and 7(D) show graphs when the moving 
distance Lact is 20 [pixel] . Figs. 7(E) and 7(F) show graphs 
when the moving distance Lact is 30 [pixel] . 

As shown in Fig. 7 , as actual moving distance of image 
capturing object (Lact) is lengthened, the distance between 
sub-pixel tracks is more uniformly subdivided, and uniform and 
fine sub-pixel motion can be captured. Namely, as the maximum 
value Amax(Lact) of the distance between sub-pixel tracks A 
and the difference between the maximum value and the minimum 

value of the distance between sub-pixel tracks A 
(Amax (Lact ) -Amin (Lact ) ) are close to 0, sub-pixel motion 
suitable for the super-resolution processing can be captured 
if moving direction at that time is selected. 

When moving distance of image capturing object (Lact) 
is determined, minimum value of Amax (Lact) is determined. 
Namely, no scan finer than the minimum value of the determined 

Amax (Lact) can be performed even when moving direction is 
adjusted in any way. The minimum value of Amax (Lact) is 1/Lact . 
For example, as shown in Fig. 7, in the case of Fig. 7(B) , the 

minimum value of Amax (Lact) is 1/10 = 0.1. In the case of Fig. 
7(D), the minimum value of Amax (Lact) is 1/20=0.05. In the 
case of Fig. 7(F), the minimum value of Amax (Lact) is 1/30. 

There are several moving directions which can obtain the 
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minimum value of Amax (Lact ) , i.e., moving direction forgiving 
Amax(Lact) close to the minimum. A moving direction for 
minimizing Amax (Lact ) -Amin (Lact ) is selected from these 
moving directions. The minimum value of 

Amax (Lact ) -Amin (Lact) is 0. 

Therefore, the following equation 5 is considered as 
normalized evaluation value at which Amax (Lact) and 
Amax (Lact) -Amin (Lact) are considered. 
[Equation 5] 

A max(Lact) x (A max(Lact) - A min(Lact)) 



Cover(Lact) = 



1 



Lact 2 

The sub-pixel motion suitable for the super-resolution 
processing of high precision is obtained by using an evaluation 
function Cover (Lact) which is represented in above equation 
5 and selecting moving direction of Cover (Lact) < 1. Namely, 
moving direction in which evaluation function Cover (Lact) ^ 
1 is formed is not selected. 

Fig. 8(A) is a graph showing the relation between 
evaluation function Cover (Lact) and moving direction when 
moving distance Lact is 30 [pixel]. In Fig. 8(A), when the 
moving directions corresponding to B and C indicated by arrows 
are selected, sub-pixel motion distributions are respectively 
shown in Figs. 8(B) and 8(C). In this connection, in Figs. 
8 (B) and 8 (C) , the moving directions are about the same and 
only values of evaluation function Cover (Lact) are different. 
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Since Cover (Lact) ^ 0 of Fig. 8(B) is formed, it is well known 
that uniform and fine sub-pixel track is entirely obtained. 
On the other hand, since Cover (Lact) =? 2 is formed in the case 
of Fig. 8 (C) even in the same moving direction, it is also well 
known that uniform and fine sub-pixel track cannot be entirely 
obtained. 

With respect to Amax(Lact) and Amin(Lact), no 
calculation equation can be analytically shown. However, for 
example, Amax(Lact) and Amin(Lact) can be calculated by a 
simple numerical calculation algorithm as shown in Fig. 9. 

<3> Optimum sub-pixel motion with respect to a minimum 
number of captured image required in the super-resolution 
processing of predetermined magnification 

Next, a concrete embodiment of the present invention will 
be explained when the super-resolution processing of 
predetermined magnification is performed. Here, minimum 
number of captured image and optimum sub-pixel motion are shown 
when the super-resolution processing of 4X4 is performed. 

In the super-resolution processing of 4X4, it is 
sufficient to obtain sub-pixel displacement every 0.25 [pixel] . 

Namely, minimum value of Amax (Lact ) =0 . 25=1/4 = 1/Lact is formed . 
Accordingly, since Lact=4 is formed, it is considered to 
measure movement of displacement of 4 [pixel] in horizontal 
direction . 
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Fig. 10(B) is a graph showing the relation between 
evaluation function Cover (4) and moving direction when moving 
distance Lact is 4 [pixel]. As can be seen from Fig. 10(B), 
there are two moving directions for forming Cover(4)=0 (i.e., 
two moving directions for obtaining sub-pixel motion suitable 
for the super-resolution processing) . That is to say, one 
moving direction is set to 1/4, and another moving direction 
is set to 3/4. 

Here, 1/4 is selected as moving direction. When moving 
direction is set to 1/4 and moving distance of image capturing 
object (Lact) is set to 4, sub-pixel motion distribution as 
shown in Fig. 10(A) is obtained. It is possible to capture 
the minimum number of captured image (16 in this example) 

required in the super-resolution processing of 4X4 from the 
sub-pixel motion distribution as shown in Fig. 10(A). 

For example, when image capturing ob j ect is moved by 0.25 
[pixel] for 1/30 [second] in horizontal direction, necessary 
time for capturing 16 images required in the super-resolution 
processing of 4X4. is about 0.5 [second] . If only appropriate 
moving direction and moving distance of image capturing object 
are determined by using the present invention, sub-pixel 
motion images of minimum number required in the 
super-resolution processing of predetermined magnification 
can be simply captured for a short time. 



23 



<4> Decision method of moving direction in which the 
amount of information relating to sub-pixel position is 
increased as the number of captured image is increased 

Conventional super-resolution processing is performed 
with resolution of predetermined magnification, e.g., a 
magnification of four times in horizontal direction and a 
magnification of four times in vertical direction. In the 
present invention, decision method of moving direction for 
obtaining sub-pixel motion suitable for the super-resolution 
processing gradually raised in resolution as in 2X2 , 3X3 
and 4X4 as capturing of image capturing object which is 
one-dimensionally moved along a predetermined moving 
direction is advanced will be considered as follows. 

Namely, in the present invention, moving direction for 
sequentially detailing sub-pixel motion suitable for the 
super-resolution processing able to sequentially utilize 
captured images as image capturing is advanced is determined 
by evaluation function SCover(LM) described later. Namely, 
moving direction for setting value of evaluation function 
SCover(LM) to be close to 0 is selected. Images obtained by 
capturing the image capturing object which is moved along 
selected moving direction is utilized. Thus, it is also 
possible to construct super-resolution processing system for 
setting image to gradually have high precision and high 
resolution . 
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When moving direction (i.e., gradient of moving 
direction) a=q/p=l/4 is set, it is optimal to set moving 
distance of image capturing object (Lact) (i.e., horizontal 
moving distance of image capturing obj ect) to 4 . However, even 
when moving distance of image capturing object is further 
increased, moving track of image capturing object merely 
passes through the same position, and no information amount 
relating to a different sub-pixel position is increased. 

Therefore, evaluation function represented in the 
following equation 6 with respect to all moving distances in 
which Lact=4 or more will be considered. 
[Equation 6] 

LM 

^ Cover {Lact) 
SCover(LM) = JM — 

Lact = A 

Here, LM indicates entire moving distance of image 
capturing object from the beginning of capturing of image 
capturing object to its termination. Namely, LM indicates 
maximum moving distance of image capturing object. Further, 
realistic application is considered and the following area 
limit is set to reduce calculating amount with respect to moving 
direction a in which 0<a<l is set to a precondition. 

Namely, first, since minimum moving distance of image 
capturing object is Lact=4, it is not necessary to consider 
an area of a ^1/4. Further, it is also not necessary to 
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consider an area of a>0.5, since a>0.5 is a repeat of 0<a<0.5. 
Accordingly, the range of 0.25<a<0.5 is considered. 

Fig. 11 shows the relation between evaluation function 
SCover(LM) and moving direction a in the case of different 
maximum moving distance LM. The maximum moving distance LM 
of Fig. 11(A) is 10 [pixel], and the maximum moving distance 
LM of Fig. 11(B) is 20 [pixel] . The maximum moving distance 
LM of Fig. 11 (C) is 40 [pixel] , and the maximum moving distance 
LM of Fig. 11(D) is 80 [pixel] . The maximum moving distance 
LM of Fig. 11(E) is 160 [pixel]. 

As can be seen from Fig. 11, for example, when maximum 
moving distance LM is 10 [pixel] , no moving direction for 
sequentially obtaining the sub-pixel motion suitable for the 
super-resolution processing of higher precision is clear so 
much. However, for example, when graph in which maximum moving 
distance LM is 160 [pixel] is seen, moving direction for 
reducing the value of SCover(LM) becomes clear. 

Such a moving direction for reducing the value of 
Scover (LM) is a good moving direction in which sub-pixel motion 
suitable for the super-resolution processing of higher 
precision is sequentially obtained as capturing is advanced. 
In short, moving direction for reducing the value of Scover (LM) 
is selected on the premise that maximum moving distance able 
to move image capturing object is entirely 160 [pixel] . Thus, 
even when moving distance of image capturing object is an 



arbitrary distance of 4 [pixel] or more and 160 [pixel] or less, 
sub-pixel motion suitable for the super-resolution processing 
is obtained. 

In reality, moving direction for setting the value of 
evaluation function SCover(LM) to be close to 0, e.g., moving 
directions of 2650/10000, 2764/10000, 2957/10000, 3037/10000, 
3866/10000, 4142/10000, 4190/10000, 4393/10000, 4488/10000, 
etc. exist. Horizontal moving distance of image capturing 
object is increased and the distance between sub-pixel tracks 
is also gradually reduced by selecting such moving directions. 
Accordingly, since information obtained with respect to 
sub-pixel displacement is also increased, it is possible to 
capture image able to be utilized in the super-resolution 
processing of high precision gradually detailed. 

For example, when moving direction a is 2957/10000, as 
shown in Fig. 12, the distance between sub-pixel tracks is 
gradually reduced and sub-pixel track is also gradually 
increased as moving distance of image capturing object (Lact) 
is increased. Namely, finer sub-pixel motion distribution is 
entirely uniformly obtained. 

<5> Moving speed of image capturing object 
Appropriate moving speed of image capturing object 
one-dimensionally moved will be considered as follows. 

The moving track of image capturing object formed when 
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image is captured at a limitlessly short time interval is 
premised so far. However, in reality, image is captured at 
an interval of 1/30 [second] in image capturing device, e.g., 
a CCD camera. Accordingly, it is necessary to limitlessly 
reduce moving speed of image capturing object to set the same 
condition, which is not realistic. In the following 
description, moving speed of image capturing object according 
to an object will be considered. 

First, in the case of capturing minimum number of 
captured image required in the super-resolution processing of 
predetermined magnification within the shortest time, image 
capturing object is controlled to a constant horizontal moving 
speed v H [pixel/sec]. At this time, moving speed of image 
capturing object along moving direction v [pixel/sec] is 
provided as shown in the following equation 7. 
[Equation 7 ] 



When frame rate of image capturing device, e.g., frame 
rate of CCD camera is set to FPS [1/sec] , it is sufficient to 
set distance A to the condition as shown in the following 
equation 8. 
[Equation 8] 




A = 



v 



H 



P 



FPS 
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If image capturing object is moved at moving speed of 
horizontal direction determined on the basis of the above 
equation 8, sub-pixel motion as shown in e.g., Fig. 13 can be 
captured. The sub-pixel motion shown in Fig . 13(A) is suitable 
for the super-resolution processing of 4X4, and the sub-pixel 
motion shown in Fig. 13 (B) is suitable for the super-resolution 
processing of 10X10. 

When it is next desirous to sequentially obtain precise 
sub-pixel motion, moving speed of image capturing object is 
changed in accordance with time, and image capturing object 
is moved at moving speed gradually slowed. For example, a 
detailed sub-pixel motion can be captured in accordance with 
time as shown in Fig. 14 by setting horizontal moving speed 

of image capturing object at time t (v H (t) [pixel/sec]) to 
the following equation 9. 
[Equation 9] 



" {4 2 ) 
a = -0.0005 

The following equation 10 is used in Fig. 14. 
[Equation 10] 

g = 2957 
p 10000 

<6> Experimental results of super-resolution processing 
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Here, sequential images of image capturing object which 
is one-dimensionally moved in a predetermined moving direction 
were captured by using a fixed single chip type CCD color camera, 
and super-resolution processing experiment of 4X4 was 
implemented by using captured Bayer image data. The image 
capturing object used in the super-resolution processing 
experiment is a magazine article stuck to a flat screw feed 
type precise moving base. The screw feed type precise moving 
base was manually moved. 

A direct super-resolution processing technique based on 
Bayer image data disclosed in patent literature 1 was utilized 
in the super-resolution processing. Further, "A 

multiparameter high precision concurrent estimation method in 
image subpixel matching " disclosed in patent literature 2 was 
utilized in movement and measurement of image capturing object. 
One high resolution image is synthesized as a result obtained 
by performing the super-resolution processing by utilizing 64 
low resolution images captured by single chip type CCD color 
camera . 

In the first super-resolution processing experiment, 
the super-resolution processing is performed by utilizing 
sequential images (64 images in total) captured by setting an 
appropriate moving direction of image capturing object which 
is determined by the present invention. Its details are shown 
in Fig. 15. 
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Fig. 15(A) shows image capturing object stuck to the 
screw feed type precise moving base. The sub-pixel motion 
image is captured by a fixed single chip type CCD color camera 
while image capturing object (i.e., a portion surrounded by 
a rectangular shape within Fig. 15(A)) is moved by a 
predetermined moving distance (Lact=107.3 [pixel] ) in moving 
direction (a=14 . 2/107 . 3) shown by an arrow within Fig. 15(A) . 
Fig. 15(B) shows a sub-pixel motion distribution obtained by 
measuring sub-pixel motion image captured by image capturing 
device for capturing an image at an interval of 1/30 [second] . 
On the other hand, the sub-pixel motion distribution can be 
calculated from moving direction of image capturing object 
(a=14 . 2/107 . 3) decided by the present invention. The 
calculated sub-pixel motion distribution is shown in Fig. 
15 (C) . 

The super-resolution processing is performed by using 
64 low resolution images (see Fig. 15 (E) ) obtained by capturing 
image capturing object surrounded by rectangular shape of Fig. 
15(D) by using single chip type CCD color camera. Thus, one 
high resolution image is obtained as shown in Fig. 15(F) . As 
can be seen from Fig. 15(F), when an appropriate moving 
direction is decided by the present invention, the result of 
the super-resolution processing is very excellent. 

In the second super-resolution processing experiment, 
when moving direction of image capturing object is 
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inappropriately set and moving distance of image capturing 
object (Lact) is insuf f icient , the super-resolution 
processing is performed by utilizing captured sequential 
images (64 images in total). Its details are shown in Fig. 
16. 

Fig. 16(A) shows image capturing object stuck to the 
screw feed type precise moving base. The sub-pixel motion 
image is captured by a fixed single chip type CCD color camera 
while image capturing object (i.e., a portion surrounded by 
a rectangular shape within Fig. 16(A)) is moved by a 
predetermined moving distance (Lact=54.1 [pixel]) in moving 
direction (a=-0 . 7 /54 . 1 ) shown by an arrow within Fig. 16(A) . 
Fig. 16(B) shows a sub-pixel motion distribution obtained by 
measuring sub-pixel motion image captured by image capturing 
device for capturing an image at an interval of 1/30 [second] . 
On the other hand, the sub-pixel motion distribution can be 
calculated from moving direction of image capturing object 
(a=-0 . 7/54 . 1) . The calculated sub-pixel motion distribution 
is shown in Fig. 16(C) . It is well known from Figs. 16(B) and 
16(C) that no sub-pixel motion is uniformly filled in areas. 

The super-resolution processing is performed by using 
64 low resolution images (see Fig. 16(E)) obtained by capturing 
image capturing object surrounded by rectangular shape of Fig. 
16(D) by using single chip type CCD color camera. Thus, the 
super-resolution processing result as shown in Fig. 16(F) is 
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obtained. As can be seen from Fig. 16(F), when moving 
direction of image capturing object is inappropriately set and 
moving distance of image capturing object (Lact) is 
insufficient, it is well known that no appropriate sub-pixel 
motion distribution can be obtained in the super-resolution 
processing and resultant image of the super-resolution 
processing is unsuccessful. 

<7> Other constructional examples 

As mentioned above, the present invention is disclosed 
in entire construction for capturing image capturing object 
while image capturing device is fixed as it is and image 
capturing object is moved in a predetermined moving direction. 
However, the present invention is not limited to such a 
construction, but can be implemented by another construction 
described later (i.e., a construction for fixing image 
capturing object as it is) . 

In the present invention, decision method of 
one-dimensional moving direction for capturing the sub-pixel 
motion image suitable for the super-resolution processing is 
preferably implemented by software using a computer system. 

<7-l> One dimensional movement of image capturing 
element within image capturing device 

In an image capturing system constructed by image 
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capturing device and image capturing object, the method for 
directly moving image capturing object in predetermined moving 
direction while fixing image capturing device as it is as 
mentioned above is considered to one-dimensionally move the 
image capturing object in the predetermined moving direction. 
In addition to this, a method for moving image capturing element 
(e.g., CCD image capturing element) within image capturing 
device (e.g., CCD camera) in the predetermined moving 
direction while fixing image capturing object is also 
considered. 

When two-dimensional drive is performed as in image 
capturing device of conventional system of the above (a) , two 
actuators are required. However, in the method according to 
the present invention, it is sufficient to perform the 
one-dimensional drive. Accordingly, moving mechanism of 
image capturing element can be constructed by one actuator. 
Therefore, image capturing device able to simply capture 
sub-pixel motion image suitable for the super-resolution 
processing can be manufactured at low cost. 

As actuator for one-dimensionally driving image 
capturing element (e.g., CCD image capturing element) within 
image capturing device (e.g., CCD camera), for example, there 
are a method for driving screw feed mechanism by rotating motor 
(see Fig. 17) , a method for utilizing an element able to realize 
very small displacement such as a piezoelectric element (see 
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Fig. 18) , etc. 



<7-2> Change "of image capturing direction of image 
capturing device 

In image capturing system constructed by image capturing 
device and image capturing object, it is possible to realize 
that image capturing object is one-dimensionally moved in a 
predetermined moving direction by changing image capturing 
direction of image capturing device while image capturing 
object is fixed. 

For example, as shown in Fig. 19, existing image 
capturing device (camera) and existing tripod are utilized, 
and an adapter (i.e., a moving direction setting adapter within 
Fig. 19) for setting horizontal direction rotating axis of 
camera (pan axis) to a constant angle (this angle is equal to 
predetermined moving direction of image capturing object, i.e. , 
a moving angle) is arranged between camera and head of tripod. 
Thus, it is possible to realize that image capturing object 
is one-dimensionally moved in the predetermined moving 
direction . 

In this moving direction setting adapter, an adapter able 
to automatically move image capturing direction of camera by 
mounting driving mechanism, and an adapter for manually 
changing moving direction of camera without mounting driving 
mechanism are considered. 
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<7-3> One-dimensional movement of moving stage 
In microscope image capturing and macro (enlargement) 
image capturing, for example, as shown in Fig. 20, it is 
possible to realize one-dimensional movement of image 
capturing object in predetermined moving direction by fixing 
image capturing device (e.g., camera) to tripod, and providing 
a mechanism able to one-dimensionally move a moving stage in 
which image capturing object is set. 

<7-4> Movement of image by using optical system within 
image capturing device 

Further, in image capturing system constructed by image 
capturing device and image capturing object, it is possible 
to realize that image capturing object is one-dimensionally 
moved in the predetermined moving direction by movement of 
image by using optical system within image capturing device 
while image capturing object is fixed. 

For example, as shown in Fig. 21, a member for optically 
moving image is built between lens and image capturing element 
(e.g., CCD image capturing element) within image capturing 
device (e.g., CCD camera). The image can be moved in a 
predetermined moving direction by controlling this member. 
For example, a prism able to control a refracting direction 
so as to control vertical angle of the prism, a shift lens, 
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etc. can be utilized as such a member. 



INDUSTRIAL APPLICABILITY 

As mentioned above, by using movement decision method 
for capturing sub-pixel motion image suitable for the 
super-resolution processing and image capturing device using 
this movement decision method according to the present 
invention, an excellent effect able to simply capture 
two-dimensional sub-pixel motion image suitable for the 
super-resolution processing for a short time by 
one-dimensional movement of image capturing object or image 
capturing element can obtain. 

By using image capturing device according to the present 
invention, an image having a displacement required in the 
super-resolution processing can be reliably captured by 
one-dimensionally moving image capturing element in the 
predetermined moving direction. The two-dimensional 

sub-pixel motion image suitable for the super-resolution 
processing captured by conventional expensive image capturing 
device having moving mechanism for two-dimensionally moving 
image capturing element with high precision can be also simply 
captured by cheap image capturing device having 
one-dimensional moving mechanism according to the present 
invention . 

Further, if image capturing object or image capturing 
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element is moved along moving direction determined on the basis 
of evaluation function Cover (Lact) of the present invention, 
an optimum two-dimensional sub-pixel motion image with respect 
to minimum number of captured image required in the 
super-resolution processing of predetermined magnification 
can be captured within a short time. 

Further, if image capturing object or image capturing 
element is moved along moving direction determined on the basis 
of evaluation function SCover(LM) of the present invention, 
the two-dimensional sub-pixel motion image for performing the 
super-resolution processing of higher precision can be simply 
captured as the number of captured image is increased. 
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